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Abstract 

Background: Missing outcome data or the inconsistent reporting of outcome data in clinical research can affect 
the quality of evidence within a systematic review. A potential solution is an agreed standardized set of outcomes 
known as a core outcome set (COS) to be measured in all studies for a specific condition. We investigated the 
amount of missing patient data for primary outcomes in Cochrane systematic reviews, and surveyed the 
Co-ordinating Editors of Cochrane Review Groups (CRGs) on issues related to the standardization of outcomes in 
their CRGs reviews. These groups are responsible for the more than 7,000 protocols and full versions of Cochrane 
Reviews that are currently available, and the several hundred new reviews published each year, presenting the 
world's largest collection of standardized systematic reviews in health care. 

Methods: Using an unselected cohort of Cochrane Reviews, we calculated and presented the percentage of 
missing patient data for the primary outcome measure chosen for each review published by each CRG. We also 
surveyed the CRG Co-ordinating Editors to see what their policies are with regards to outcome selection and 
outcomes to include in the Summary of Finding (SoF) tables in their Cochrane Reviews. They were also asked to list 
the main advantages and challenges of standardizing outcomes across all reviews within their CRG. 

Results: In one fifth of the 283 reviews in the sample, more than 50% of the patient data for the primary outcome 
was missing. Responses to the survey were received from 90% of Co-ordinating Editors. Thirty-six percent of CRGs 
have a centralized policy regarding which outcomes to include in the SoF table and 73% of Co-ordinating Editors 
thought that a COS for effectiveness trials should be used routinely for a SoF table. 

Conclusions: The reliability of systematic reviews, in particular meta-analyses they contain, can be improved if 
more attention is paid to missing outcome data. The availability of COSs for specific health conditions might help 
with this and the concept has support from the majority of Co-ordinating Editors in CRGs. 

Keywords: COMET, Cochrane review groups, Co-ordinating editors, Core outcome set, ORBIT, Systematic reviews, 
Survey 



Background collection of systematic reviews, with more than 5,000 full 

Cochrane Reviews are systematic reviews of research in reviews and protocols for another 2,000 published to date, 
human health care and health policy, and are internation- Outcome selection in systematic reviews needs to be rele- 
ally recognized as a high quality source of evidence for de- vant to patients, clinicians and policy-makers if the find- 
cision making. They bring together research evidence on ings of a review are to influence practice and future 
the effects of healthcare interventions or the accuracy of research. Inconsistent choice of outcome measures in clin- 
diagnostic tests into the world's largest, standardized ical trials means that many meta-analyses are unable to in- 
clude data from all the relevant studies. For example, the 
five most accessed Cochrane Reviews in 2009, together 
with the top cited review in that year, all described incon- 
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hampered the ability of the reviewers to resolve the uncer- 
tainties about health care that they set out to tackle [1]. 
An additional problem that can affect the quality of evi- 
dence within a systematic review is that of missing out- 
come data from all eligible trials. Missing outcome data in 
trials can occur for a number of reasons. Participant data 
can be missing as a result of attrition or as a result of the 
selective non-reporting of an outcome in a study. Recent 
research has shown that outcome reporting bias (ORB), 
that is, results-based selection for publication of a subset 
of the original outcome variables, is also a major problem 
in randomized trials [2], The Outcome Reporting Bias in 
Trials (ORBIT) study of the impact of ORB in randomized 
trials on the results of Cochrane Reviews found that ORB 
is an 'under-recognised problem that affects the conclu- 
sions in a substantial proportion of Cochrane reviews' [3]. 
This study found that more than half (157/283 (55%)) of 
the reviews did not include full data for the review's pri- 
mary outcome of interest from all eligible trials. The me- 
dian amount of review primary outcome data missing for 
any reason was 10%, whereas 50% or more of the potential 
data were missing in 70 (25%) reviews. In this same study, 
an assessment of the impact of ORB was undertaken for 
reviews with a single meta-analysis of the primary review 
outcome where there was missing outcome data for a 
whole study that was known or suspected of being mea- 
sured but not reported. Of the 25 reviews that met these 
criteria, a sensitivity analysis revealed that the conclusions 
to eight of these reviews were not robust to ORB — that is, 
the treatment effect estimate changed from a significant 
result favoring treatment to a non-significant result. For 
all the 25 reviews assessed, the median percentage change 
in the treatment effect estimates after the adjustment 
based on the sensitivity analysis was 39% (interquartile 
range 18% to 67%) [3]. 

One way to reduce the amount of missing outcome 
data by ensuring that eligible trials contribute usable in- 
formation is the definition and implementation of an 
agreed minimum set of standardized outcomes, to be 
measured and reported in all trials for a particular dis- 
ease or condition, referred to as a 'Core Outcome Set' 
(COS) [4,5]. Furthermore, some Cochrane Review 
Groups (CRGs) now include a Summary of Findings 
(SoF) table in their reviews [6]. The SoF table presents 
the results of the review for up to seven outcomes that 
are important to patients and aims to improve the 
understanding and retrieval of key findings. 

In this paper, we present results for missing data in 
single review primary outcomes from the Cochrane 
Reviews included in the ORBIT study. We also report 
the results of a survey of the Co-ordinating Editors of 
CRGs, undertaken to obtain their views on the benefits 
and challenges of core outcome sets and how they might 
improve the quality of the reviews in their CRG. We 



were also interested in establishing whether the amount 
of missing data from a systematic review was influenced 
by whether or not a CRG had a centralized policy on 
outcome selection. The Co-ordinating Editors have re- 
sponsibility for overseeing the work of their CRG and 
deciding on the publication of each of the Cochrane 
Reviews from within their CRG. 

Methods 

Missing data 

The missing study data reported here was part of the lar- 
ger ORBIT project to estimate the prevalence and effect 
of outcome reporting bias in clinical trials on the pri- 
mary outcomes of systematic reviews from three issues 
of The Cochrane Library: issue 4, 2006; issue 1, 2007, 
and issue 2, 2007. For the 283 reviews in this study, the 
number of participants in each of the eligible studies 
was extracted and the percentage of missing participant 
data was calculated as the number of participants miss- 
ing from the meta-analyses of the single review primary 
outcome (as a result of the non-reporting of outcome 
data in a study), as defined by the lead reviewer. Data 
from reviews containing at least five eligible studies are 
presented. 

Survey of Co-ordinating Editors of Cochrane Review 
Groups 

With the exception of the Cochrane Methodology Re- 
view Group (which focuses on topics relating to method- 
ology, rather than healthcare interventions and tests), 
the Co-ordinating Editors for the 50 CRGs (as of 2011) 
were contacted and asked to participate in a short online 
survey with regards to how a COS might improve the 
quality of the reviews in their CRG. The survey was con- 
structed using the online survey software, SurveyGizmo, 
and was open for responses between 1 November 2011 
and 31 July 2012. Non-responders were contacted peri- 
odically if a response to the survey was not received. As 
part of the survey, Co-ordinating Editors were provided 
with a brief synopsis of how COSs could make it easier 
for their review authors to identify appropriate outcomes 
for Cochrane Reviews, and present their findings clearly 
and succinctly. They were also presented with the miss- 
ing data plot for all CRGs and made aware of the data 
relating to their CRG. Co-ordinating Editors were asked 
whether or not they had any centralized CRG policies 
relating to the outcomes to include in their reviews (in- 
cluding the distinction between primary and secondary 
outcomes) and outcomes to include in the SoF tables 
(providing one is included in the review), or whether 
these decisions are left to the authors of the review. Pat- 
terns in the amount of missing participant data between 
CRGs with and without a centralized policy on outcome 
selection were examined. Finally, Co-ordinating Editors 
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were asked if they thought that a COS for effectiveness 
trials should be used routinely for a SoF table and what 
they thought were the main advantages and challenges 
of standardizing outcomes across all reviews in a par- 
ticular condition covered by their CRG. All advantages 
and challenges were independendy reviewed by two authors 
(JJK and PRW) and categorized into common topics. Any 
discrepancies were resolved through discussion. 

Results 

Of the 283 reviews included in the ORBIT study, 143, 
from 40 CRGs, were included here, that is, reviews that 
contained at least 5 eligible studies (Figure 1). Forty-one 
(29%) reviews contained no missing patient data, while 
26 (18%) had more than 50% of the patient participant 
data on the single primary outcome missing from the re- 
view. Accountable reasons for missing data included the 
difficulty to measure certain outcomes, the diverse range 
of possible outcomes to measure and disputes around 
the importance and best outcomes to measure which are 
both robust and reliable for use in systematic reviews. A 
summary of the relevant responses with regards to the 
reasons for missing outcome data that was provided by 
the Co-ordinating Editors is provided in Table 1. 

Responses to the survey were received from Co- 
ordinating Editors for 45 (90%) of the 50 CRGs. Fourteen 
Co-ordinating Editors (31%) had previously been involved 
in the development of a COS and 16 (36%) were aware of 
other work that has been done or is ongoing to develop a 
COS for conditions relevant to their CRG. Overall, 16 
(36%) CRGs have a centralized policy for outcome selec- 
tion while the remaining 29 leave outcome selection and 
the relative importance of outcomes chosen to the 



discretion of the authors of the relevant review. From 
Figure 1, there did not appear to be any association be- 
tween the existence of a CRG policy for outcome selection 
and the amount of missing data in their reviews. Similarly, 
16 (36%) CRGs have a centralized policy regarding which 
outcomes to include in the SoF table (14 of which had a 
centralized policy for outcome selection), while 25 (56%) 
left this decision to the review authors and a further four 
(9%) are not yet including the SoF table in their reviews. 
All four CRGs where the SoF has not yet been adopted 
had no CRG policy for outcome selection but two of the 
four Co-ordinating Editors were aware of ongoing COS 
development work for conditions covered by their CRG. 

A total of 33 (73%) Co-ordinating Editors thought that a 
COS for effectiveness trials should be used routinely for a 
SoF table. Reasons for this were provided by 23 Co- 
ordinating Editors and included relevant outcomes mea- 
sured and reported (10), comparability of outcomes (6), 
improved interpretability of outcomes (4), standardization 
of outcomes (2) and reduces risk of bias (1). Ten of the 12 
Co-ordinating Editors who did not think a COS should be 
used in the SoF table thought that the scope and diversity 
of their reviews would not allow a COS to be applied 
across the reviews for their CRG. 

The Co-ordinating Editors listed 100 advantages and 82 
challenges associated with standardizing outcomes across 
reviews in their CRG (Table 2). The most common advan- 
tage for having a COS was that systematic reviews or 
meta-analyses would benefit from the standardization of 
outcomes. The main challenges related to the process 
needed to develop a COS, decisions around when a COS 
should be applied and the need to persuade potential 
authors of Cochrane Reviews to use the COS. 



100 



0 



■ Cochrane Review WITHOUT CRG outcome policy 
° Cochrane Review WITH CRG outcome policy 



Cochrane Review Group 

Figure 1 The percentage of missing patient data for the primary outcome for each review that had at least five included studies 
(reviews first published in Issue 4, 2006 to Issue 2, 2007). 
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Table 1 Responses from Cochrane review group co-editors who provided reasons for missing data in their reviews 

Cochrane Review Group Explanation from Cochrane Review Group Co-Editor 

Pain, Palliative Care and 
Support Care 

Dementia and Cognitive 
mpairment 



Oral Health 

Neuromuscular Diseases 



Renal 



Cystic Fibrosis and Genetic 
Disorders 

Heart 



Wounds 



1 think this is about right [amount of missing data]. Our scope is large and these problems are more common when 
there are no received methods (e.g. rehabilitation, palliative care, complex interventions) and less common where 
there is a history of methods development and consensus (e.g., headache). 

'Would suggest that it may not be possible in our CRG to specify outcomes only by the condition without reference 
to the type of intervention. For example, some interventions in dementia may aim to modify the course of the 
disease, and cognitive outcomes should clearly be part of a minimum outcome set in such trials; for others, such as 
certain psychosocial interventions (e.g. touch and massage), modifying cognitive decline would be an unreasonable 
aim, and collection of cognitive data would be unduly burdensome for patients and researchers'. 

'Our reviews are very diverse so we would need to develop many different core outcome groups'. 

'Neuromuscular Disease is very diverse, even sometimes within one disease. For instance some patients with Motor 
Neuron Disease will have lots of spasticity and complications related to that, others flaccidity and others bulbar 
involvement. Variations in outcomes chosen depend upon the focus of the intervention and the researcher. We try to 
keep our outcomes fairly loosely defined - e.g. any measure of walking ability - especially in physio/fatigue/strength 
training and other qualitative research areas - but even then there are some fairly abstract outcomes measured that 
don't fit in clearly'. 

The problem is the heterogeneity of our scope. Unlike some other CRGs we are an organ-related CRG (and then 
cover many orphan reviews). We cover everything from UTI (urinary tract infections) to transplantation'. 

'One set of outcomes is not always appropriate for every trial in cystic fibrosis, e.g. for a nutritional intervention body 
mass index may be appropriate. For a respiratory therapy it may be spirometry'. 

The issue of missing data for primary outcomes may reflect the biomedical bias in many heart disease trials - so if we 
have 20 RCTs in a review with a clinical primary outcome specified, it is very likely that only a few of the trials will 
have reported on the clinical outcome but many more will have reported on physiological outcomes....Our scope is 
much wider than many review groups, so I can envisage difficulties coming up with a core outcome set'. 

'Our reviews cover a variety of conditions and interventions (probably most groups do) and relate to topics with quite 
under-developed research cultures (nursing, dermatology, surgery) so the high levels of missing data in our field 
doesn't surprise me'. 



Discussion 

There is considerable variability within and between 
CRGs in the amount of data that are missing for the pri- 
mary outcomes chosen for a review. However, there did 
not appear to be any association between the existence 
of a CRG policy for outcome selection and the amount 
of missing data in their reviews (Figure 1). Furthermore, 
the survey of Co-ordinating Editors revealed a clear lack 
of clarity and consensus on the outcomes to measure 
within the specialties covered by CRGs. 

The set of reviews that are reported in this study are 
slightly dated. However, the reviews used came from the 
earlier ORBIT study which started in 2006. The missing 
participant data from the primary outcome from each 
review was extracted during the ORBIT study, although 



Advantage for a systematic review/meta-analysis 39 

Improves interpretation/guidance 19 

Outcome likely to be more appropriate 16 

Advantage for the design of a new study 13 

Improves something about the outcome itself 6 

Reduces outcome reporting bias 6 

Reduces resource requirement 1 



this remained unpublished data. Presenting the data here 
accompanied the Cochrane Co-Editor survey data well 
and demonstrated the importance of COSs from a 
reviewer perspective. A recent inspection of a random 
sample of 20 of the 283 reviews used in ORBIT revealed 
that only 25% (5/20) of these reviews identified new 
studies for inclusion in the most updated version of the 
review - this meant on average, in at least 75% of cases, 
the percent missing data from the reviews used in 2006/ 
2007 still remains unchanged. 

The suggestion that a COS should be routinely used as 
part of a SoF table was well received, with most Co- 
ordinating Editors believing that this would be beneficial. 
The scope and diversity of reviews within some CRGs is 
likely to be a barrier to the application of a COS across 



n 

Development of a COS 23 

Something about scope 21 

How to persuade authors/trialists/industry to implement 20 

'How' to measure once the 'what' has been decided 1 1 

Important outcomes not currently being measured 2 

Resource to develop 2 

Updating process 2 

Conflict of interest 1 



Table 2 Advantages and challenges of standardizing outcomes across all reviews for a particular condition 

Advantages n Challenges 
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the reviews for the particular group. It was noted by 
some of the Co-ordinating Editors in these CRGs that 
outcome specification for particular conditions would be 
difficult without further refinement to specific popula- 
tions and interventions (for example, pharmacological 
versus non-pharmacological treatments). 

The main advantage of a COS as suggested by the ma- 
jority of Co-ordinating Editors was that the concept of a 
COS would benefit the systematic review process by in- 
creasing the amount of usable information for use in a 
meta-analysis. The challenges that were identified appear 
to reflect a lack of experience with COS, which could be 
overcome with training or support materials (for example 
on how to develop a COS and how to encourage others to 
use it). To help with this, the Core Outcome Measures in 
Effectiveness Trials (COMET) Initiative brings together 
researchers interested in the development and application 
of COS (www.comet-initiative.org). COMET aims to col- 
late and stimulate relevant resources (both applied and 
methodological), facilitate the exchange of ideas and infor- 
mation, and foster methodological research. The Initiative 
has three strategic goals: to increase the number of COSs 
developed using evidence-based methods; to increase their 
impact on the quality of research by raising awareness and 
increasing their use; and to establish methods for the 
development of COS [5]. This will include work with Co- 
ordinating Editors and others in The Cochrane Collabor- 
ation to facilitate activities within CRGs to develop COS 
and to implement these within Cochrane Reviews, includ- 
ing the provision of training workshops and the prepar- 
ation of training materials. This work should also have 
benefits for systematic reviewers outside of The Cochrane 
Collaboration, who wish to maximize the impact of COS 
on the quality of reviews and the research studies that they 
bring together. 



each review was extracted by JJK, who produced the missing data plots for 
each CRG. JJK, EG and MC sent invitations and reminders to Co-ordinating 
Editors to complete the survey. JJK prepared the initial manuscript. PRW and 
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before submission. 
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Conclusions 

The reliability of systematic reviews, in particular meta- 
analyses they contain, can be improved if more attention 
is paid to missing outcome data. The availability of 
COSs for specific health conditions might help with this 
and the concept has support from the majority of Co- 
ordinating Editors in CRGs. 
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